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Abstract 


Background and Context 

The national high school graduation rate reached its highest level in U.S. history—82 percent— 
during the 2013/14 school year (Kena et al., 2016)—but dropout remains a persistent problem in 
the Midwest and nationally. Early warning systems that use research-based warning signs to identify 
students at risk of dropping out have emerged as one strategy for improving graduation rates. These 
warning signs can include indicators of engagement, behavior, and course performance during middle 
and high school (Allensworth & Easton, 2005, 2007; Balfanz, Herzog, & Mac Iver, 2007; Silver, 
Saunders, & Zarate, 2008). More robust, comprehensive early warning systems also emphasize 
matching and assigning identified students with interventions to help them get on track for graduation 
(Neild, Balfanz, & Herzog, 2007) and continued monitoring of students’ progress in these 
interventions (O’Cummings, & Therriault, 2015; O’Cummings, Therriault, Heppen, Yerhot, & 
Hauenstein, 2011). 


However, little is known about the impact of the use of early warning systems on students and 
schools. This study examined the impact and implementation of one early warning system—the Early 
Warning Intervention and Monitoring System (EWIMS)—on student and school outcomes in three 
Midwestern states. 


Research Questions 
The study examined the following research questions after the first year of EWIMS adoption: 


1. What is the impact of EWIMS on indicators of student risk? 
2. What is the impact of EWIMS on student progress in school? 


To provide context for the impact findings, the study also documented schools’ implementation of 
EWIMS. 


Intervention 

EWIMS is a systematic approach to using data to identify students who are at risk of not graduating 
on time, assigning interventions to students flagged as at risk, and monitoring their response to 
intervention. EWIMS provides schools guidance and site-based support to implement a seven-step 
process, supported by use of an early warning data tool (see Figure 1). The tool incorporates validated 
indicators of attendance, course performance, and behavior, based on prior research (see Table 1), to 
flag students who are at risk of not graduating on time (Heppen & Therriault, 2008; Therriault, 
Heppen, O’Cummings, Fryer, & Johnson, 2010) and to allow schools to assign students to 
interventions and monitor progress through multiple reporting features. The EWIMS model intends to 
help schools efficiently and effectively use data to identify at-risk students and provide targeted 
supports to get students back on track for graduation (see Figure 2 for the theory of action). 


Research Design, Setting, and Sample Sizes 


The study included 73 high schools from three Midwest states that were randomly assigned within 
matched pairs to implement EWIMS during the 2014-15 school year (37 EWIMS schools) or 
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continue business as usual (36 control schools).1 Students in Grades 9 and 10 were the focus of the 
study (with a total of 37,671 students; 18,634 students in treatment and 19,037 students in control). 
There were no statistically significant differences between treatment and control schools on any 
measured baseline characteristics. 


Data Collection and Measurement 

Extant student records were collected from the 2012—13 school year through spring 2015 from 
both treatment and control schools. Student outcomes were binary variables coded 1 or 0, 
reflecting whether the student was above or below the threshold for each risk indicator (missed 
10 percent or more of instructional time, failed one or more courses, grade point average [GPA] 
of 2.0 or lower, and one or more suspensions) or, for progress in school, whether the student 
earned sufficient credits to be on track to graduate within four years. 


Implementation measures included extant documents from the 2014—15 school year, monthly logs 
of content and frequency of EWIMS team meetings, reports from the early warning data tool that 
measured tool use, and interviews with EWIMS team members. 


Analytic Methods 

Multilevel logistic and linear regression models with students (Level 1) nested in schools (Level 
2) and a treatment indicator at Level 2 were used to estimate the impact of EWIMS on students. 
Student-level covariates and fixed effects for matched pairs were included in these models to 
increase the precision of the impact estimates. 


Preliminary Findings and Results 
Findings (see Figure 3) show that: 


e The percentage of students who were chronically absent (that is, missed 10 percent or 
more of instructional time) was lower in EWIMS schools (10 percent) than in control 
schools (14 percent). This 4 percentage point difference is statistically significant. 

e The percentage of students who failed one or more courses was lower in EWIMS schools 
(21 percent) than in control schools (26 percent). This 5 percentage point difference is 
statistically significant. 

e There was no statistically significant difference in the percentage of students who had a GPA 
of 2.0 or lower between EWIMS schools (17 percent) and control schools (19 percent) or in 
the percentage of students who were suspended once or more (9 percent in both). 


EWIMS also did not have an impact on student progress in school (having insufficient credits). 


The impact of EWIMS on chronic absence and course failures was larger for the Grade 9 cohort 
than the Grade 10 cohort. 


Implementation of the EWIMS seven-step process was low, and implementing EWIMS was 
challenging for participating schools. 


" Control schools were offered EWIMS during the following school year (2015-16). 
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Conclusions 

This study provides rigorous initial evidence that, even with limited implementation during the 
first year of adoption, using a comprehensive early warning system can reduce the percentage of 
students who are chronically absent or fail one or more courses. These short-term results are 
promising because chronic absence and course failures in Grades 9 and 10 are two key indicators 
that students are off track for graduation. The study also provides evidence that EWIMS is 
challenging to implement in the first year of adoption. Future research should focus on the 
mechanisms through which EWIMS had an impact on chronic absences and course failures, why 
it did not affect other outcomes, and whether (and how) after more time or with stronger 
implementation, the observed impacts fade, grow larger, or expand to the other outcomes. 
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Appendix B. Tables and Figures 


Table 1. Definitions of the Flags Used by the EWIMS Tool to Identify Students at Risk for High School 
Dropout 


Flag Definition 


Chronic absence flag One flag for the first 20 or 30 days, one flag per grading 
period, and a cumulative flag for the year: Missing 10 
percent of instructional time. 


Course failure flag One flag per grading period and a cumulative flag for 
the year: Failed one or more semester or trimester 
courses in any subject. 


Grade-point average flag One flag per grading period and a cumulative flag for 
the year: Earned a 2.0 or lower on a 4.0 scale or 
equivalent on a different scale. 


Behavior flag One flag per grading period and a cumulative flag for 
the year: Suspended once or more. 


Off-track flag One cumulative flag for year: Failed two or more 
semester-long or three or more trimester-long core 
courses (English, math, science, social studies) or 
accumulated fewer credits than required for promotion 
to the next grade. 


Source. EWIMS Implementation Guide. For more information about EWIMS implementation, see http://www.earlywarningsyste ms.org/wp- 
content/uploads/documents/EWSHSImplementationguide2013.pdf or Therriault et al. (2010). 
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Figure 1. The EWIMS Seven-Step Implementation Process 


Step 7: Step 1: 
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Refine the Roles and 
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Review the 
Early Warning 
Data 
Step 4: 
Interpret the 
Early 
Warning 
Data 


Source: EWIMS Implementation Guide. For more information about EWIMS implementation, see http://www.earlywarningsystems.org/wp- 
content/uploads/documents/EWSHSImplementationguide2013.pdf or Therriault et al. (2010). 
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Figure 2. Theory of Action for How the Early Warning Intervention and Monitoring System Improves 
School and Student Outcomes 


Intermediate Long-term 
outcomes outcome 


¢ Adopt EWIMS |j* EWIMS team _ je Reduced Increased ¢ Improved on- 
systematic identifies percentage of percentage of time high school 
approach students as at risk] students at risk || students graduation rates 


e Seven-step using research- due to chronic progressing and 


process based indicators absence, course persisting in 


: of on-time failure, low school 
¢ Early warning 


data tool graduation cumulative GPA, |]e Improved school 


* Flagged students and suspensions |} data culture 
are assigned 

supports and 

interventions 

* EWIMS team 

monitors student 

progress in 

interventions 


Source: Authors’ theory. 


Figure 3. Compared to Students in Control schools, a Lower Percentage of Students in 
EWIMS Schools Were at Risk Due to Chronic Absence and Course Failure, but Not Due to 
Low GPA or Behavior, at the End of the 2014-15 School Year 


Model-adjusted HEWIMS schools m Control schools 
pete ntage 
100% 
80% 
60% 
KK 
40% OK 
26% 
14% a 17%, 19% 
| mull = 
Chronic absence Failed any course Low GPA Suspended 


GPA is grade point average. 
*** Significant at p < .001. 


Note: This figure presents model-adjusted percentage of students identified as at-risk in EWIMS and control schools, controlling for school 
and student covariates. Higher values indicate a larger percentage of students at risk. 


Sample included 65 schools and 35,888 students for chronic absence; 65 schools and 35,133 students for failed any course; 57 schools 
and 30,080 students for low GPA; and 63 schools and 35,501 students for suspended. 
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